{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2019 Google Inc. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#      http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# =============================================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train and deploy Xgboost (Scikit-learn) on Kubeflow from Notebooks\n",
    "\n",
    "This notebook introduces you the usage of Kubeflow Fairing to train and deploy a model to Kubeflow on Google Kubernetes Engine (GKE), and Google Cloud AI Platform training. This notebook demonstrate how to:\n",
    " \n",
    "* Train an XGBoost model in a local notebook,\n",
    "* Use Kubeflow Fairing to train an XGBoost model remotely on Kubeflow cluster,\n",
    "* Use Kubeflow Fairing to train an XGBoost model remotely on AI Platform training,\n",
    "* Use Kubeflow Fairing to deploy a trained model to Kubeflow, and Call the deployed endpoint for predictions.\n",
    "\n",
    "**You need Python 3.6 to use Kubeflow Fairing.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setups\n",
    "\n",
    "* Pre-conditions\n",
    "    - Deployed a kubeflow cluster through https://deploy.kubeflow.cloud/\n",
    "    - Have the following environment variable ready: \n",
    "        - PROJECT_ID # project host the kubeflow cluster or for running AI platform training\n",
    "        - DEPLOYMENT_NAME # kubeflow deployment name, the same the cluster name after delpoyed\n",
    "        - GCP_BUCKET # google cloud storage bucket\n",
    "\n",
    "* Create service account\n",
    "```bash\n",
    "export SA_NAME = [service account name]\n",
    "gcloud iam service-accounts create ${SA_NAME}\n",
    "gcloud projects add-iam-policy-binding ${PROJECT_ID} \\\n",
    "    --member serviceAccount:${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com \\\n",
    "    --role 'roles/editor'\n",
    "gcloud iam service-accounts keys create ~/key.json \\\n",
    "    --iam-account ${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com\n",
    "```\n",
    "\n",
    "* Authorize for Source Repository\n",
    "```bash\n",
    "gcloud auth configure-docker\n",
    "```\n",
    "\n",
    "* Update local kubeconfig (for submiting job to kubeflow cluster)\n",
    "```bash\n",
    "export CLUSTER_NAME=${DEPLOYMENT_NAME} # this is the deployment name or the kubenete cluster name\n",
    "export ZONE=us-central1-c\n",
    "gcloud container clusters get-credentials ${CLUSTER_NAME} --region ${ZONE}\n",
    "```\n",
    "\n",
    "* Set the environmental variable: GOOGLE_APPLICATION_CREDENTIALS\n",
    "```bash\n",
    "export GOOGLE_APPLICATION_CREDENTIALS = ....\n",
    "```\n",
    "```python\n",
    "os.environ['GOOGLE_APPLICATION_CREDENTIALS']=...\n",
    "```\n",
    "\n",
    "* Install the lastest version of fairing\n",
    "```python\n",
    "pip install git+https://github.com/kubeflow/fairing@master\n",
    "```\n",
    "\n",
    "* Upload training file\n",
    "```bash\n",
    "# upload the train.csv to GCS bucket that can be accessed from both CMLE and Kubeflow cluster\n",
    "gsutil cp ./train.csv ${GCP_Bucket}/train.csv\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please not that the above configuration is required for notebook service running outside Kubeflow environment. And the examples demonstrated in the notebook is fully tested on notebook service outside Kubeflow cluster also.**\n",
    "\n",
    "**The environemt variables, e.g. service account, projects and etc, should have been pre-configured while setting up the cluster.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up your notebook for training an XGBoost model\n",
    "\n",
    "Import the libraries required to train this model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import argparse\n",
    "import logging\n",
    "import joblib\n",
    "import sys\n",
    "import pandas as pd\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.impute import SimpleImputer\n",
    "from xgboost import XGBClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.basicConfig(format='%(message)s')\n",
    "logging.getLogger().setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a \"quota exceeded\" or \"API not enabled\" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/\n",
      "  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import fairing\n",
    "\n",
    "# Setting up google container repositories (GCR) for storing output containers\n",
    "# You can use any docker container registry istead of GCR\n",
    "# For local notebook, GCP_PROJECT should be set explicitly\n",
    "GCP_PROJECT = fairing.cloud.gcp.guess_project_name()\n",
    "GCP_Bucket = os.environ['GCP_BUCKET'] # e.g., 'gs://kubeflow-demo-g/'\n",
    "\n",
    "# This is for local notebook instead of that in kubeflow cluster\n",
    "# os.environ['GOOGLE_APPLICATION_CREDENTIALS']="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the model logic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a function to split the input file into training and testing datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gcs_copy(src_path, dst_path):\n",
    "    import subprocess\n",
    "    print(subprocess.run(['gsutil', 'cp', src_path, dst_path], stdout=subprocess.PIPE).stdout[:-1].decode('utf-8'))\n",
    "    \n",
    "def gcs_download(src_path, file_name):\n",
    "    import subprocess\n",
    "    print(subprocess.run(['gsutil', 'cp', src_path, file_name], stdout=subprocess.PIPE).stdout[:-1].decode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_input(source_path, test_size=0.25):\n",
    "    \"\"\"Read input data and split it into train and test.\"\"\"\n",
    "    \n",
    "    file_name = source_path.split('/')[-1]\n",
    "    gcs_download(source_path, file_name)\n",
    "    data = pd.read_csv(file_name)\n",
    "    data.dropna(axis=0, inplace=True)\n",
    "\n",
    "    y = data.Class\n",
    "    X = data.drop(['Class', 'Amount', 'Time'], axis=1).select_dtypes(exclude=['object'])\n",
    "\n",
    "    train_X, test_X, train_y, test_y = train_test_split(X.values,\n",
    "                                                      y.values,\n",
    "                                                      test_size=test_size,\n",
    "                                                      shuffle=True)\n",
    "\n",
    "    imputer = SimpleImputer()\n",
    "    train_X = imputer.fit_transform(train_X)\n",
    "    test_X = imputer.transform(test_X)\n",
    "\n",
    "    return (train_X, train_y), (test_X, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define functions to train, evaluate, and save the trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(train_X,\n",
    "                train_y,\n",
    "                test_X,\n",
    "                test_y,\n",
    "                n_estimators,\n",
    "                learning_rate):\n",
    "    \"\"\"Train the model using XGBRegressor.\"\"\"\n",
    "    model = XGBClassifier(n_estimators=n_estimators, learning_rate=learning_rate)\n",
    "\n",
    "    model.fit(train_X,\n",
    "            train_y,\n",
    "            early_stopping_rounds=40,\n",
    "            eval_set=[(test_X, test_y)])\n",
    "\n",
    "    print(\"Best loss on eval: %.2f with %d rounds\",\n",
    "               model.best_score,\n",
    "               model.best_iteration+1)\n",
    "    return model\n",
    "\n",
    "def eval_model(model, test_X, test_y):\n",
    "    \"\"\"Evaluate the model performance.\"\"\"\n",
    "    predictions = model.predict_proba(test_X)\n",
    "    logging.info(\"auc=%.2f\", roc_auc_score(test_y, predictions[:,1]))\n",
    "\n",
    "def save_model(model, model_file):\n",
    "    \"\"\"Save XGBoost model for serving.\"\"\"\n",
    "    joblib.dump(model, model_file)\n",
    "    gcs_copy(model_file, GCP_Bucket + model_file)\n",
    "    logging.info(\"Model export success: %s\", model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a class for your model, with methods for training and prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FraudServe(object):\n",
    "    \n",
    "    def __init__(self):\n",
    "        self.train_input = GCP_Bucket + \"train_fraud.csv\"\n",
    "        self.n_estimators = 50\n",
    "        self.learning_rate = 0.1\n",
    "        self.model_file = \"trained_fraud_model.joblib\"\n",
    "        self.model = None\n",
    "\n",
    "    def train(self):\n",
    "        (train_X, train_y), (test_X, test_y) = read_input(self.train_input)\n",
    "        model = train_model(train_X,\n",
    "                          train_y,\n",
    "                          test_X,\n",
    "                          test_y,\n",
    "                          self.n_estimators,\n",
    "                          self.learning_rate)\n",
    "\n",
    "        eval_model(model, test_X, test_y)\n",
    "        save_model(model, self.model_file)\n",
    "\n",
    "    def predict(self, X, feature_names):\n",
    "        \"\"\"Predict using the model for given ndarray.\"\"\"\n",
    "        if not self.model:\n",
    "            self.model = joblib.load(self.model_file)\n",
    "        # Do any preprocessing\n",
    "        prediction = self.model.predict(data=X)\n",
    "        # Do any postprocessing\n",
    "        return [[prediction.item(0), prediction.item(0)]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train an XGBoost model in a notebook\n",
    "\n",
    "Call `FraudServe().train()` to train your model, and then evaluate and save your trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[0]\tvalidation_0-error:0.037534\n",
      "Will train until validation_0-error hasn't improved in 40 rounds.\n",
      "[1]\tvalidation_0-error:0.024129\n",
      "[2]\tvalidation_0-error:0.021448\n",
      "[3]\tvalidation_0-error:0.021448\n",
      "[4]\tvalidation_0-error:0.021448\n",
      "[5]\tvalidation_0-error:0.02681\n",
      "[6]\tvalidation_0-error:0.032172\n",
      "[7]\tvalidation_0-error:0.02681\n",
      "[8]\tvalidation_0-error:0.02681\n",
      "[9]\tvalidation_0-error:0.029491\n",
      "[10]\tvalidation_0-error:0.02681\n",
      "[11]\tvalidation_0-error:0.018767\n",
      "[12]\tvalidation_0-error:0.021448\n",
      "[13]\tvalidation_0-error:0.018767\n",
      "[14]\tvalidation_0-error:0.018767\n",
      "[15]\tvalidation_0-error:0.018767\n",
      "[16]\tvalidation_0-error:0.018767\n",
      "[17]\tvalidation_0-error:0.018767\n",
      "[18]\tvalidation_0-error:0.018767\n",
      "[19]\tvalidation_0-error:0.018767\n",
      "[20]\tvalidation_0-error:0.018767\n",
      "[21]\tvalidation_0-error:0.021448\n",
      "[22]\tvalidation_0-error:0.018767\n",
      "[23]\tvalidation_0-error:0.021448\n",
      "[24]\tvalidation_0-error:0.024129\n",
      "[25]\tvalidation_0-error:0.021448\n",
      "[26]\tvalidation_0-error:0.021448\n",
      "[27]\tvalidation_0-error:0.018767\n",
      "[28]\tvalidation_0-error:0.018767\n",
      "[29]\tvalidation_0-error:0.018767\n",
      "[30]\tvalidation_0-error:0.018767\n",
      "[31]\tvalidation_0-error:0.018767\n",
      "[32]\tvalidation_0-error:0.018767\n",
      "[33]\tvalidation_0-error:0.021448\n",
      "[34]\tvalidation_0-error:0.021448\n",
      "[35]\tvalidation_0-error:0.021448\n",
      "[36]\tvalidation_0-error:0.021448\n",
      "[37]\tvalidation_0-error:0.024129\n",
      "[38]\tvalidation_0-error:0.024129\n",
      "[39]\tvalidation_0-error:0.024129\n",
      "[40]\tvalidation_0-error:0.024129\n",
      "[41]\tvalidation_0-error:0.024129\n",
      "[42]\tvalidation_0-error:0.024129\n",
      "[43]\tvalidation_0-error:0.024129\n",
      "[44]\tvalidation_0-error:0.024129\n",
      "[45]\tvalidation_0-error:0.024129\n",
      "[46]\tvalidation_0-error:0.024129\n",
      "[47]\tvalidation_0-error:0.02681\n",
      "[48]\tvalidation_0-error:0.02681\n",
      "[49]\tvalidation_0-error:0.02681\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "auc=0.99\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best loss on eval: %.2f with %d rounds 0.018767 12\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model export success: trained_fraud_model.joblib\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "FraudServe().train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make Use of Fairing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spicify a image registry that will hold the image built by fairing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In this demo, I use gsutil, therefore i compile a special image to install GoogleCloudSDK as based image\n",
    "base_image = 'gcr.io/{}/fairing-predict-example:latest'.format(GCP_PROJECT)\n",
    "!docker build --build-arg PY_VERSION=3.6.4 . -t {base_image}\n",
    "!docker push {base_image}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "DOCKER_REGISTRY = 'gcr.io/{}/fairing-job-xgboost'.format(GCP_PROJECT)\n",
    "BASE_IMAGE = base_image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train an XGBoost model remotely on Kubeflow\n",
    "\n",
    "Import the `TrainJob` and `GKEBackend` classes. Kubeflow Fairing packages the `FraudServe` class, the training data, and the training job's software prerequisites as a Docker image. Then Kubeflow Fairing deploys and runs the training job on Kubeflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using preprocessor: <class 'fairing.preprocessors.function.FunctionPreProcessor'>\n",
      "Using docker registry: gcr.io/gojek-kubeflow/fairing-job-xgboost\n",
      "Using builder: <class 'fairing.builders.docker.docker.DockerBuilder'>\n",
      "Building the docker image.\n",
      "Building image using docker\n",
      "Docker command: ['python', '/app/function_shim.py', '--serialized_fn_file', '/app/pickled_fn.p']\n",
      "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...\n",
      "Creating docker context: /tmp/fairing_context_dip8f0em\n",
      "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...\n",
      "Context: /tmp/fairing_context_dip8f0em, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py at /app/fairing/__init__.py\n",
      "Context: /tmp/fairing_context_dip8f0em, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/runtime_config.py at /app/fairing/runtime_config.py\n",
      "Context: /tmp/fairing_context_dip8f0em, Adding requirements.txt at /app/requirements.txt\n",
      "Context: /tmp/fairing_context_dip8f0em, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/functions/function_shim.py at /app/function_shim.py\n",
      "Context: /tmp/fairing_context_dip8f0em, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/__init__.py at /app/cloudpickle/__init__.py\n",
      "Context: /tmp/fairing_context_dip8f0em, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/cloudpickle.py at /app/cloudpickle/cloudpickle.py\n",
      "Context: /tmp/fairing_context_dip8f0em, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmpw4wpdn9j at /app/pickled_fn.p\n",
      "Context: /tmp/fairing_context_dip8f0em, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmpcnblrwou at /app/HousingServe.py\n",
      "Context: /tmp/fairing_context_dip8f0em, Adding /tmp/fairing_dockerfile_1mwzspba at Dockerfile\n",
      "Building docker image gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job:C0F59817...\n",
      "Build output: Step 1/7 : FROM gcr.io/gojek-kubeflow/fairing-predict-example:latest\n",
      "Build output: \n",
      "Build output: ---> 07b0c0a773a2\n",
      "Build output: Step 2/7 : WORKDIR /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> e38aad2dc182\n",
      "Build output: Step 3/7 : ENV FAIRING_RUNTIME 1\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 597bd070338a\n",
      "Build output: Step 4/7 : COPY /app//requirements.txt /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 05e78d5eb908\n",
      "Build output: Step 5/7 : RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> e31aa3ffcc59\n",
      "Build output: Step 6/7 : COPY /app/ /app/\n",
      "Build output: \n",
      "Build output: ---> 2ad6caba9fab\n",
      "Build output: Step 7/7 : CMD python /app/function_shim.py --serialized_fn_file /app/pickled_fn.p\n",
      "Build output: \n",
      "Build output: ---> Running in 22adbd7dcf8c\n",
      "Build output: ---> 3837c52fb3b9\n",
      "Push finished: {'ID': 'sha256:3837c52fb3b9bb8545fcbddb83b4c6b7ecaca5a6d01d7a3637ccd803c84dbad3'}\n",
      "Build output: Successfully built 3837c52fb3b9\n",
      "Build output: Successfully tagged gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job:C0F59817\n",
      "Publishing image gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job:C0F59817...\n",
      "Push output: The push refers to repository [gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job] None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Pushing [>                                                  ]     512B/47.07kB\n",
      "Push output: Pushing [==================================================>]  56.32kB\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Pushed None\n",
      "Push output: C0F59817: digest: sha256:fe0da235a59bdea843178a21920be6056f466926abd850c20c52c52374ad3d04 size: 3473 None\n",
      "Push finished: {'Tag': 'C0F59817', 'Digest': 'sha256:fe0da235a59bdea843178a21920be6056f466926abd850c20c52c52374ad3d04', 'Size': 3473}\n",
      "Not able to find gcp credentials secret: user-gcp-sa\n",
      "Training job fairing-job-ppql8 launched.\n",
      "Waiting for fairing-job-ppql8-2wk8f to start...\n",
      "Pod started running True\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying gs://kubeflow-demo-g/train.csv...\n",
      "/ [1 files][449.9 KiB/449.9 KiB]                                                \n",
      "Operation completed over 1 objects/449.9 KiB.\n",
      "\n",
      "[0]\tvalidation_0-rmse:177514\n",
      "Will train until validation_0-rmse hasn't improved in 40 rounds.\n",
      "[1]\tvalidation_0-rmse:161858\n",
      "[2]\tvalidation_0-rmse:147237\n",
      "[3]\tvalidation_0-rmse:134132\n",
      "[4]\tvalidation_0-rmse:122224\n",
      "[5]\tvalidation_0-rmse:111538\n",
      "[6]\tvalidation_0-rmse:102142\n",
      "[7]\tvalidation_0-rmse:93392.3\n",
      "[8]\tvalidation_0-rmse:85824.6\n",
      "[9]\tvalidation_0-rmse:79667.6\n",
      "[10]\tvalidation_0-rmse:73463.4\n",
      "[11]\tvalidation_0-rmse:68059.4\n",
      "[12]\tvalidation_0-rmse:63350.5\n",
      "[13]\tvalidation_0-rmse:59732.1\n",
      "[14]\tvalidation_0-rmse:56260.7\n",
      "[15]\tvalidation_0-rmse:53392.6\n",
      "[16]\tvalidation_0-rmse:50770.8\n",
      "[17]\tvalidation_0-rmse:48107.8\n",
      "[18]\tvalidation_0-rmse:45923.9\n",
      "[19]\tvalidation_0-rmse:44154.2\n",
      "[20]\tvalidation_0-rmse:42488.1\n",
      "[21]\tvalidation_0-rmse:41263.3\n",
      "[22]\tvalidation_0-rmse:40212.8\n",
      "[23]\tvalidation_0-rmse:39089.1\n",
      "[24]\tvalidation_0-rmse:37691.1\n",
      "[25]\tvalidation_0-rmse:36875.2\n",
      "[26]\tvalidation_0-rmse:36276.2\n",
      "[27]\tvalidation_0-rmse:35444.1\n",
      "[28]\tvalidation_0-rmse:34831.5\n",
      "[29]\tvalidation_0-rmse:34205.4\n",
      "[30]\tvalidation_0-rmse:33831.9\n",
      "[31]\tvalidation_0-rmse:33183.6\n",
      "[32]\tvalidation_0-rmse:33019.4\n",
      "[33]\tvalidation_0-rmse:32680\n",
      "[34]\tvalidation_0-rmse:32438.5\n",
      "[35]\tvalidation_0-rmse:32130.4\n",
      "[36]\tvalidation_0-rmse:31644.2\n",
      "[37]\tvalidation_0-rmse:31248.9\n",
      "[38]\tvalidation_0-rmse:31059.8\n",
      "[39]\tvalidation_0-rmse:30862.4\n",
      "[40]\tvalidation_0-rmse:30754\n",
      "[41]\tvalidation_0-rmse:30561.6\n",
      "[42]\tvalidation_0-rmse:30416.9\n",
      "[43]\tvalidation_0-rmse:30156.4\n",
      "[44]\tvalidation_0-rmse:29852.9\n",
      "[45]\tvalidation_0-rmse:29486.6\n",
      "[46]\tvalidation_0-rmse:29158.8\n",
      "[47]\tvalidation_0-rmse:29017\n",
      "[48]\tvalidation_0-rmse:28973.9\n",
      "[49]\tvalidation_0-rmse:28787.7\n",
      "mean_absolute_error=18173.15\n",
      "Copying file://trained_ames_model.joblib [Content-Type=application/octet-stream]...\n",
      "AccessDeniedException: 403 Insufficient Permission                              \n",
      "Model export success: trained_ames_model.joblib\n",
      "Best RMSE on eval: %.2f with %d rounds 28787.720703 50\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cleaning up job fairing-job-ppql8...\n"
     ]
    }
   ],
   "source": [
    "from fairing import TrainJob\n",
    "from fairing.backends import GKEBackend\n",
    "\n",
    "train_job = TrainJob(FraudServe, BASE_IMAGE, input_files=[\"requirements.txt\"],\n",
    "                     docker_registry=DOCKER_REGISTRY, backend=GKEBackend())\n",
    "train_job.submit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train an XGBoost model remotely on Cloud ML Engine\n",
    "\n",
    "Import the `TrainJob` and `GCPManagedBackend` classes. Kubeflow Fairing packages the `FraudServe` class, the training data, and the training job's software prerequisites as a Docker image. Then Kubeflow Fairing deploys and runs the training job on Cloud ML Engine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using preprocessor: <class 'fairing.preprocessors.function.FunctionPreProcessor'>\n",
      "Using docker registry: gcr.io/gojek-kubeflow/fairing-job-xgboost\n",
      "Using builder: <class 'fairing.builders.docker.docker.DockerBuilder'>\n",
      "Building the docker image.\n",
      "Building image using docker\n",
      "Docker command: ['python', '/app/function_shim.py', '--serialized_fn_file', '/app/pickled_fn.p']\n",
      "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...\n",
      "Creating docker context: /tmp/fairing_context_t8_nr827\n",
      "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...\n",
      "Context: /tmp/fairing_context_t8_nr827, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py at /app/fairing/__init__.py\n",
      "Context: /tmp/fairing_context_t8_nr827, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/runtime_config.py at /app/fairing/runtime_config.py\n",
      "Context: /tmp/fairing_context_t8_nr827, Adding requirements.txt at /app/requirements.txt\n",
      "Context: /tmp/fairing_context_t8_nr827, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/functions/function_shim.py at /app/function_shim.py\n",
      "Context: /tmp/fairing_context_t8_nr827, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/__init__.py at /app/cloudpickle/__init__.py\n",
      "Context: /tmp/fairing_context_t8_nr827, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/cloudpickle.py at /app/cloudpickle/cloudpickle.py\n",
      "Context: /tmp/fairing_context_t8_nr827, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmpi9ejaors at /app/pickled_fn.p\n",
      "Context: /tmp/fairing_context_t8_nr827, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmp8aqop6nc at /app/HousingServe.py\n",
      "Context: /tmp/fairing_context_t8_nr827, Adding /tmp/fairing_dockerfile_7yyuw6a7 at Dockerfile\n",
      "Building docker image gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job:11CD9CDA...\n",
      "Build output: Step 1/7 : FROM gcr.io/gojek-kubeflow/fairing-predict-example:latest\n",
      "Build output: \n",
      "Build output: ---> 07b0c0a773a2\n",
      "Build output: Step 2/7 : WORKDIR /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> e38aad2dc182\n",
      "Build output: Step 3/7 : ENV FAIRING_RUNTIME 1\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 597bd070338a\n",
      "Build output: Step 4/7 : COPY /app//requirements.txt /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 05e78d5eb908\n",
      "Build output: Step 5/7 : RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> e31aa3ffcc59\n",
      "Build output: Step 6/7 : COPY /app/ /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 2ad6caba9fab\n",
      "Build output: Step 7/7 : CMD python /app/function_shim.py --serialized_fn_file /app/pickled_fn.p\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 3837c52fb3b9\n",
      "Push finished: {'ID': 'sha256:3837c52fb3b9bb8545fcbddb83b4c6b7ecaca5a6d01d7a3637ccd803c84dbad3'}\n",
      "Build output: Successfully built 3837c52fb3b9\n",
      "Build output: Successfully tagged gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job:11CD9CDA\n",
      "Publishing image gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job:11CD9CDA...\n",
      "Push output: The push refers to repository [gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job] None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: 11CD9CDA: digest: sha256:fe0da235a59bdea843178a21920be6056f466926abd850c20c52c52374ad3d04 size: 3473 None\n",
      "Push finished: {'Tag': '11CD9CDA', 'Digest': 'sha256:fe0da235a59bdea843178a21920be6056f466926abd850c20c52c52374ad3d04', 'Size': 3473}\n",
      "file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py\", line 36, in autodetect\n",
      "    from google.appengine.api import memcache\n",
      "ModuleNotFoundError: No module named 'google.appengine'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py\", line 33, in <module>\n",
      "    from oauth2client.contrib.locked_file import LockedFile\n",
      "ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py\", line 37, in <module>\n",
      "    from oauth2client.locked_file import LockedFile\n",
      "ModuleNotFoundError: No module named 'oauth2client.locked_file'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py\", line 41, in autodetect\n",
      "    from . import file_cache\n",
      "  File \"/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py\", line 41, in <module>\n",
      "    'file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth')\n",
      "ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth\n",
      "URL being requested: GET https://www.googleapis.com/discovery/v1/apis/ml/v1/rest\n",
      "URL being requested: POST https://ml.googleapis.com/v1/projects/gojek-kubeflow/jobs?alt=json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating training job with the following options: {'jobId': 'fairing_job_7df6e38a', 'trainingInput': {'scaleTier': 'BASIC', 'masterConfig': {'imageUri': 'gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job:11CD9CDA'}, 'region': 'us-central1'}}\n",
      "Job submitted successfully.\n",
      "Access job logs at the following URL:\n",
      "https://console.cloud.google.com/mlengine/jobs/fairing_job_7df6e38a?project=gojek-kubeflow\n"
     ]
    }
   ],
   "source": [
    "from fairing import TrainJob\n",
    "from fairing.backends import GCPManagedBackend\n",
    "train_job = TrainJob(FraudServe, BASE_IMAGE, input_files=[\"requirements.txt\"],\n",
    "                     docker_registry=DOCKER_REGISTRY, backend=GCPManagedBackend())\n",
    "train_job.submit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the trained model to Kubeflow for predictions\n",
    "\n",
    "Import the `PredictionEndpoint` and `KubeflowGKEBackend` classes. Kubeflow Fairing packages the `FraudServe` class, the trained model, and the prediction endpoint's software prerequisites as a Docker image. Then Kubeflow Fairing deploys and runs the prediction endpoint on Kubeflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This part only works for fairing version >=0.5.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using preprocessor: <class 'fairing.preprocessors.function.FunctionPreProcessor'>\n",
      "Using docker registry: gcr.io/gojek-kubeflow/fairing-job-xgboost\n",
      "Using builder: <class 'fairing.builders.docker.docker.DockerBuilder'>\n",
      "Building the docker image.\n",
      "Building image using docker\n",
      "Docker command: ['python', '/app/function_shim.py', '--serialized_fn_file', '/app/pickled_fn.p']\n",
      "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...\n",
      "Creating docker context: /tmp/fairing_context_yonwsrbc\n",
      "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...\n",
      "Context: /tmp/fairing_context_yonwsrbc, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py at /app/fairing/__init__.py\n",
      "Context: /tmp/fairing_context_yonwsrbc, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/runtime_config.py at /app/fairing/runtime_config.py\n",
      "Context: /tmp/fairing_context_yonwsrbc, Adding requirements.txt at /app/requirements.txt\n",
      "Context: /tmp/fairing_context_yonwsrbc, Adding trained_ames_model.joblib at /app/trained_ames_model.joblib\n",
      "Context: /tmp/fairing_context_yonwsrbc, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/functions/function_shim.py at /app/function_shim.py\n",
      "Context: /tmp/fairing_context_yonwsrbc, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/__init__.py at /app/cloudpickle/__init__.py\n",
      "Context: /tmp/fairing_context_yonwsrbc, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/cloudpickle.py at /app/cloudpickle/cloudpickle.py\n",
      "Context: /tmp/fairing_context_yonwsrbc, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmp7nk9d31l at /app/pickled_fn.p\n",
      "Context: /tmp/fairing_context_yonwsrbc, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmp40474wwu at /app/HousingServe.py\n",
      "Context: /tmp/fairing_context_yonwsrbc, Adding /tmp/fairing_dockerfile_tv9eejm2 at Dockerfile\n",
      "Building docker image gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job:28D8F94A...\n",
      "Build output: Step 1/7 : FROM gcr.io/gojek-kubeflow/fairing-predict-example:latest\n",
      "Build output: \n",
      "Build output: ---> 07b0c0a773a2\n",
      "Build output: Step 2/7 : WORKDIR /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> e38aad2dc182\n",
      "Build output: Step 3/7 : ENV FAIRING_RUNTIME 1\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 597bd070338a\n",
      "Build output: Step 4/7 : COPY /app//requirements.txt /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 05e78d5eb908\n",
      "Build output: Step 5/7 : RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> e31aa3ffcc59\n",
      "Build output: Step 6/7 : COPY /app/ /app/\n",
      "Build output: \n",
      "Build output: ---> 67e07b1b0768\n",
      "Build output: Step 7/7 : CMD python /app/function_shim.py --serialized_fn_file /app/pickled_fn.p\n",
      "Build output: \n",
      "Build output: ---> Running in 005665229ac6\n",
      "Build output: ---> d93b497cc2fb\n",
      "Push finished: {'ID': 'sha256:d93b497cc2fb1e2aad5df6b33b1ef1fc2fd85d9b5ac44bde28424983d0d07d8b'}\n",
      "Build output: Successfully built d93b497cc2fb\n",
      "Build output: Successfully tagged gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job:28D8F94A\n",
      "Publishing image gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job:28D8F94A...\n",
      "Push output: The push refers to repository [gcr.io/gojek-kubeflow/fairing-job-xgboost/fairing-job] None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Pushing [>                                                  ]  1.024kB/82.13kB\n",
      "Push output: Pushing [==================================================>]  92.16kB\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Pushed None\n",
      "Push output: 28D8F94A: digest: sha256:d450620b10c82d58557203ef25211b4cb0c11cc7463401c475e24d87af89e610 size: 3473 None\n",
      "Push finished: {'Tag': '28D8F94A', 'Digest': 'sha256:d450620b10c82d58557203ef25211b4cb0c11cc7463401c475e24d87af89e610', 'Size': 3473}\n",
      "Deploying the endpoint.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for prediction endpoint to come up...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cluster endpoint: http://35.239.163.72:5000/predict\n",
      "Prediction endpoint: http://35.239.163.72:5000/predict\n"
     ]
    }
   ],
   "source": [
    "from fairing import PredictionEndpoint\n",
    "from fairing.backends import KubeflowGKEBackend\n",
    "# The trained_ames_model.joblib is exported during the above local training\n",
    "endpoint = PredictionEndpoint(FraudServe, BASE_IMAGE, input_files=['trained_fraud_model.joblib', \"requirements.txt\"],\n",
    "                              docker_registry=DOCKER_REGISTRY, backend=KubeflowGKEBackend())\n",
    "endpoint.create()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy to GCP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deploy model to gcp\n",
    "# from fairing.deployers.gcp.gcpserving import GCPServingDeployer\n",
    "# deployer = GCPServingDeployer()\n",
    "# deployer.deploy(VERSION_DIR, MODEL_NAME, VERSION_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Call the prediction endpoint\n",
    "\n",
    "Create a test dataset, then call the endpoint on Kubeflow for predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "{\"data\":{\"names\":[\"t:0\",\"t:1\"],\"tensor\":{\"shape\":[1,2],\"values\":[165164.875,165164.875]}},\"meta\":{}}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "(train_X, train_y), (test_X, test_y) = read_input(GCP_Bucket + \"train_fraud.csv\")\n",
    "endpoint.predict_nparray(test_X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean up the prediction endpoint\n",
    "\n",
    "Delete the prediction endpoint created by this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Deleted service: kubeflow/fairing-service-hpxqk\n",
      "Deleted deployment: kubeflow/fairing-deployer-sntlx\n"
     ]
    }
   ],
   "source": [
    "endpoint.delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "virtualPython36",
   "language": "python",
   "name": "virtualpython36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
